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Abstract 

A  model  of  situation  assessment  is  proposed  that 
integrates  processes  of  recognition  and 
metacognition.  The  model  consists  of  memory 
structures  that  represent  what  is  believed, 
value/action  structures  that  represent  the  values 
placed  on  those  beliefs,  and  metacognitive 
processes  used  to  test,  critique,  and  correct  situation 
comprehension.  Training  implications  are  derived 
from  the  model,  and  an  ongoing  study  of 
metacognitive  skills  training  is  described. 

1.  Introduction 

The  evolution  of  Army  doctrine  and  practice 
towards  flexible  and  independent  planning  poses  a 
significant  challenge  to  existing  thinking  about 
battlefield  situation  assessment.  One  example  of  this 
trend  is  the  concept  of  battlespace,  which  is  a  spatial 
and  temporal  mental  model  composed  by  the 
individual  officer,  that  encompasses  his  command's 
capabilities  and  interests.  A  second  example  is  the 
new  emphasis  on  using  abbreviated  methods  of 
planning  in  time-stressed  situations.  These  methods 
focus  on  a  single  best  course  of  action  and  abandon  the 
exhaustive  analytical  models  previously  endorsed  for 
situations  where  time  is  scarce.  This  shift  to  flexible 
methods  and  independence  of  command  places  a 
significant  new  burden  on  officers.  It  requires  that 
they  have  skills  in  situation  assessment  and  planning 
that  previously  were  attributed  to  their  senior 
command. 

At  the  same  time  that  Army  doctrine  is  changing  to 
increase  emphasis  on  situation  assessment  skills,  recent 
cognitive  research  promise  to  leverage  our 
understanding  of  situation  assessment  and  planning, 
and  help  us  to  devise  better  training  and  decision  aids. 


The  research  to  which  we  refer  concerns  the  role  of 
recognition  and  metacognition  in  expert  performance. 

It  is  the  foundation  for  an  alternative  to  analytical  and 
recognitional  models  of  decision  making.  We  call  this 
the  adaptive  model  of  decision  making  (Cohen,  1993a). 

According  to  the  analytical  or  normative  approach, 
an  assessment  is  rational  if  it  is  founded  on  a  logically 
consistent  set  of  judgments  about  probabilities  and 
values.  Consistency  is  defined  with  respect  to  formal 
constraints  dictated  by  “self-evident”  axioms.  There  is 
ample  research  claiming  that  unaided  human  decisions 
do  not  satisfy  such  constraints.  Indeed,  systematic 
errors,  or  “biases,”  have  been  identified  at  virtually 
every  stage  of  the  decision  making  process  (Cohen, 
1993b;  Kahenman,  Slovic,  and  Tversky,  1982). 
Analytical  methods  do  not  reflect  the  way  actual  expert 
decision  makers  handle  problems,  particularly  in 
complex  situations  requiring  rapid  situation  assessment 
and  planning. 

The  recognitional  viewpoint  equates  successful 
assessment  with  a  virtually  automatic  (rather  than 
controlled)  response  to  environmental  patterns, 
sensitive  detection  of  stimuli  near  the  threshold  of 
awareness,  and  the  use  of  easily  retrieved,  task-specific 
encodings  in  skilled  memory.  A  view  of  this  kind  has 
also  appears  in  research  on  expert-novice  differences  in 
a  variety  of  domains.  Beginning  with  Chase  and 
Simon's  ( 1973)  work  on  chess,  expertise  has  been 
equated  with  mastery  of  a  large  repertoire  of  familiar 
patterns  and  their  associated  responses.  However, 
pattern-recognition  views  say  little  about  decision 
making  in  novel  problems.  How  are  schemas  updated 
and  maintained  in  new  and  changing  circumstances? 
How  are  conflicting  and  unreliable  data  dealt  with  ? 
How  do  decision  makers  change  their  minds?  It  is 
clear  that  simple  pattern-matching  and  retrieval  are  not 
the  whole  story. 
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In  this  paper,  we  explore  a  third  approach  to 
modeling  and  training  battlefield  situation  assessment. 
It  is  a  naturalistic  approach  in  which  analysis  of 
decision  evaluation  begins  with  the  way  experienced, 
effective  decision  makers  actually  make  decisions  in 
real-world  tasks  (Klein,  Orisanu,  Calderwood,  & 
Zsambok,  1993,  Holyoak,  1993).  To  this  end  we  have 
interviewed  33  Lt.  Colonels  and  Majors  responsible  for 
planning  at  the  division,  brigade,  and  battalion  levels 
concerning  their  experiences  in  assessing  battlefield 
situations.  We  have  found,  as  have  others  in  expert- 
novice  research,  that  these  officers  use  methods  that 
combine  pattern  recognition  with  strategies  for 
effectively  facilitating  recognition,  verifying  its  results, 
and  constructing  adequate  models  when  recognition 
fails.  From  this  foundation,  we  have  developed  an 


“adaptive  model'’  of  decision  making  that  integrates 
recognition  and  metacognitive  processes. 

2.  The  Model 

2.1  Overview 

The  framework  that  we  propose  for  analyzing 
battlefield  situation  assessment  framework  consists,  at 
the  most  general  level,  of  four  components  (see  figure 
#1): 

1 .  The  real-world  environment. 

2.  Memory  and  knowledge  structures; 

3.  Actions,  goals,  and  values;  and 

4.  Processes  for  regulating  and  monitoring  cognition 


Figure  1:  Framework  for  the  adaptive  decision-making  model. 


The  figure  represents  these  components  as  arcs  that 
comprise  a  circle.  The  diagram  partitions  these 
components  with  concentric  circles,  which  represent 
different  aspects  of  memory,  from  the  immediate  focus 
of  working  memory  in  the  innermost  ring,  to  the 
contents  of  long-term  semantic  and  episodic  memory 
in  the  outer  ring  (as  discussed  in  the  following 
section). 

The  basic  form  of  the  framework  is  inspired  by 
Neisser  (1976).  In  his  concept  of  the  perceptual  cycle, 
knowledge  structures  called  schemas  actively  direct 
actions  such  as  attending  to  and  exploring  the  environ¬ 
ment.  The  real-world  information  generated  by  that 
exploration  then  causes  changes  in  the  schemas.  These 
interactions  cycle  continuously  as  the  observer  gains 
understanding  of  the  actual  world.  Connolly  and 
Wagner  (1988)  extended  Neisser's  concept  to  include 
decision  cycles,  in  which  exploration  of  the  environ¬ 
ment  causes  decision  makers  to  refine  their  goals.  We 
have  incorporated  this  extension,  and  have  added  the 
iterative  role  of  metacognition,  i.e.,  monitoring  and 
regulating  one's  own  cognitive  processes,  in  learning 
both  about  the  world  and  about  one's  own  goals. 

The  operation  of  this  expanded  model  is  more 
complex  than  that  of  Neisser's  original.  Neisser's 
perceptual  cycle  comprised  only  the  sequence  from 
knowledge  to  action  to  real-world  and  back  to 
knowledge.  Our  notion  of  cognitive  cycle,  however, 
includes  many  other,  more  complex  possibilities.  The 
metacognitive  component  critiques  and  corrects 
situation  knowledge  and  plan  knowledge;  plan 
knowledge  directs  sampling  of  the  environment;  these 
perceptual  samples  modify  situation  knowledge,  which 
can  direct  the  composition  of  plan  knowledge  without 
the  mediation  of  metacognitive  processes.  As  just  one 
example  of  the  potential  cycles  of  assessment,  an  initial 
knowledge  structure  may  be  checked  by  metacognitive 
processes,  modified,  and  checked  again,  before  leading 
to  an  action  plan,  which  is  also  checked  by  metacogni¬ 
tion  before  being  implemented  in  the  environment, 
resulting  in  new  knowledge. 

To  understand  the  potential  of  the  model,  it  is 
important  to  examine  each  of  its  three  cognitive 
components  (excluding  the  environment)  more  closely. 

2.2  Situation  knowledge:  Enemy  plan  structures  & 
Action  schemas 

We  partition  knowledge  structure  in  this  model,  as 
do  Adams,  Tenney,  and  Pew  (1991),  into  four 
segments; 

1 .  The  explicit  focus  of  working  memory 
(representing  the  currently  attended  part  of  the 
situation), 

2.  The  implicit  focus  of  memory  (containing  the  full 
situation  model). 


3.  Current  episodic  memory  (containing  the  history 
of  the  current  problem),  and 

4.  Long-term  memory  (with  both  semantic  and 
episodic  contents). 

Long-term  knowledge  takes  the  form  of  memory 
schemas.  These  are  used  to  organize  situation 
assessment  information.  More  proficient  situation 
assessors  appear  to  use  a  variety  of  schemas.  Among 
the  most  common  are  the  enemy  plan  schema  and  the 
action  schema. 

Commanders  use  enemy  plan  structures  organize 
information  about  enemy  interests,  strengths,  and 
location,  and  describe  how  they  lead  to  intentions, 
actions,  and  consequences.  Take,  for  example,  a 
situation  in  which  a  U.S.  officer  must  infer  the  intent 
and  actions  of  an  enemy  who  may  attack  along 
northern  or  southern  fronts.  The  officer  knows  that 
tanks  are  a  prime  source  of  enemy  strength,  and  that 
U.S.  forces  have  situated  tank  killing  systems  in  a 
southern  region.  Enemy  interests  (a  term  we  use 
broadly  define  to  include  values,  doctrine,  and  goals) 
include  doctrine  to  avoid  opposing  strength,  such  as 
northern  U.S.  tank  killing  systems.  Finally,  enemy 
location  allows  attack  in  the  north  or  south  because 
terrain  is  hospitable  in  both  areas.  These  factors 
enable  the  U.S.  officer  to  infer  that  the  enemy  intent  is 
to  attack  to  the  north,  that  this  will  involve  such 
actions  as  moving  artillery,  massing  troops,  and 
command  facilities  to  the  north.  These  enemy’s 
intended  consequence  will  be  a  northern  breakthrough. 

The  action  schema  builds  on  the  enemy  plan 
structure.  It  describes  three  modes  of  inferring  enemy 
intent:  proactive,  predictive,  reactive. 

In  the  proactive  mode,  the  officer's  conception  of 
the  situation  is  predicated  on  molding  enemy  intent  by 
shaping  the  battlefield,  and  specifically  by  altering  the 
enemy's  perception  of  his  own  interests,  strength  or 
location.  For  example,  an  officer's  assessment  of  a 
situation  may  assume  that  his  planned  deceptions  will 
influence  enemy  estimations  of  relative  strength,  and 
that  the  deceptions  will  persuade  the  enemy  to  adopt  an 
intent  to  attack  a  force  that  is,  in  fact,  superior  to  his 
own. 

The  predictive  orientation  is  one  in  which  the 
commander  uses  his  knowledge  of  enemy  interests, 
strength  and  location  to  predict  enemy  intent.  For 
example,  the  officer  who  understands  the  enemy 
interest,  strength  and  location  described  above  might 
predict  the  northern  attack. 

In  the  reactive  mode,  a  commander  infers  enemy 
intent  from  the  actions  the  enemy  carries  out,  or  by 
observing  their  consequences.  The  commander  hit 
from  the  north  by  enemy  tanks  would,  clearly,  infer  the 
enemy's  intent,  and  possibly  reconstruct  the  causal 
chain  to  that  intent  from  enemy  interests,  strength  and 
location. 


These  strategies  are  not  mutually  exclusive.  A 
predictive  strategy  may  employ  reactive  methods  (i.e., 
observations  of  enemy  actions )  to  confirm  the 
predictions.  A  proactive  strategy  may  use  predictive 
methods  to  decide  what  actions  would  produce  the 
desired  enemy  intent,  and  may  use  reactive  methods  to 
confirm  that  the  attempt  to  influence  enemy  intent  was 
successful. 

There  are  a  range  of  other  schemas  that  officers 
bring  to  bear  on  situation  assessment.  We  will  not 
discuss  them  here,  but  the  following  list  conveys  their 
character: 

•  Enemy  goal  structures  describe  the  hierarchical 
and  compensatory  relationships  among  ultimate 
values,  principles,  goals,  and  actions. 

•  Temporal  plan  execution  structures  provide  a  more 
detailed  description  of  the  temporal  durations, 
precedence  relations,  and  causal  contingencies 
among  actions  and  events. 

•  Enemy  planning/C2  structures  describe  the  enemy 
roles  and  activities  involved  in  producing, 
communicating,  and  implementing  plans. 

•  Terrain  structures  relate  terrain  features  to 
expected  enemy  actions  and  prescribed  friendly 
actions. 

2.3  Plan/Goal  knowledge:  Value/action  structures 

Actions,  goals,  and  values  reflect  a  qualitatively 
different  way  of  viewing  knowledge.  They  represent 
how  possible  states  of  affairs  are  valued,  whereas 
situation  knowledge  represents  how  strongly  they  are 
believed.  Values  or  preferences  are  importantly 
different  from  strengths  of  belief.  They  influence 
beliefs,  but  are  separate  from  them.  We  draw  on 
Beach's  (1993)  concepts  of  the  value  image,  trajectory 
image,  and  strategic  image  to  define  these  segments. 

High-level  values  and  principles  are  relatively 
permanent  knowledge  about  what  the  decision  maker 
regards  as  desirable,  important,  and  worth  pursuing. 

Current  goals  are  episodic  memory  structures. 

Goals  are  sequences  of  desired  states;  they  are  concrete 
realizations  of  high-level  values  and  principles  in  the 
current  situation.  They  give  meaning  to  specific  plans, 
and  are  used  to  generate  plans. 

The  current  plan  is  the  detailed  set  of  actions  and 
action  contingencies  that  the  decision  maker  has 
adopted  in  the  current  situation.  It  includes  the  specific 
actions  (e.g.,  "move  up  follow-up  forces,"  "emplace 
artillery,"  look  for  a  kill  zone,  etc.)  undertaken  to 
realize  goals. 

The  part  of  the  plan  active  in  working  memory  is 
the  immediate  focus  of  evaluation.  Such  evaluation 
may  occur  prior  to  implementation  as  part  of  the 
decision  making  process,  or  during  implementation  by 
monitoring  an  on-going  action  for  its  success  in 
achieving  goals. 


Processing  of  actions,  goals  and  values  can  be 
either  top  down  or  bottom  up  (Beach,  1990).  Actions 
and  plans  may  be  generated  and  evaluated  based  on 
goals,  and  modified  or  rejected  if  they  fail  to  achieve 
them.  Similarly,  goals  may  be  generated  and  evaluated 
based  on  values.  On  the  other  hand,  from  a  bottom-up 
perspective,  goals  may  be  revised  if  no  actions  can  be 
found  to  achieve  them.  Even  high-level  values  (such  as 
avoiding  one’s  first  defeat)  might  be  revised  (perhaps 
rationalized  away)  if  they  are  not  achievable  by  realis¬ 
tic  goals  or  actions. 

2.4  Metacognitive  knowledge:  Overview 

Metacognition  consists  of  functions  that  monitor 
and  regulate  thought.  It  has  been  defined  as 
“individuals'  knowledge  of  the  states  and  processes  of 
their  own  mind  and/or  their  ability  to  control  or  modify 
these  states  and  processes”  (Gavelek  and  Raphael, 
1985). 

Metacognition  is  the  focus  of  considerable  attention 
by  developmental  psychologists  (e.g.,  Flavell,  1979; 
Forrest-Pressley,  MacKinnon,  and  Waller,  1985), 
interested  in  how  children  learn  to  manage  the 
cognitive  activities  involved  in  reading, 
comprehending,  memorizing,  and  paying  attention. 
However,  it  is  more  informative  for  the  present 
purposes  to  reference  the  expert/novice  literature. 

There  is  abundant  evidence  for  the  significance  of 
metacognition  in  expert  problem  solving.  While 
experts  may  be  said  to  “recognize”  familiar  problems, 
recognition  is  sometimes  achieved  through  the 
evaluation  of  intermediate  results.  For  example, 
according  to  Larkin,  et  al.  ( 1980),  physics  experts  often 
construct  and  examine  a  sketch  of  the  superficial 
objects  and  relations  in  a  physics  problem  in  order  to 
determine  the  next  step:  If  the  depicted  system  is 
familiar,  the  expert  may  proceed  directly  to  the 
equations  required  for  solution.  If  the  system  is 
unfamiliar,  the  expert  constructs  an  idealized 
representation  (i.e.,  a  free-body  diagram),  which  is 
then  used  in  the  generation  of  solution  equations. 
According  to  Chi,  Glaser,  and  Rees  ( 1982),  this 
qualitative  analysis  of  a  problem  is  not  a  discrete  phase 
that  is  concluded  prior  to  the  generation  of  quantitative 
equations.  They  found  that  experts  returned  to,  and 
refined,  the  initial  gross  representation  when  necessary 
throughout  the  course  of  the  problem.  In  short,  experts 
manipulate  the  situation  until  they  recognize  it.  That 
is,  they  change  their  representation  of  the  problem 
until  it  makes  contact  with  their  knowledge. 
Metacognitive  skill  is  required  in  judgments  of 
familiarity  and  of  how  best  to  transform  the  problem  to 
make  it  familiar. 

Metacognition  also  plays  a  role  after  the  problem 
has  been  recognized  and  (apparently)  “solved.”  Physics 
experts  utilize  the  abstract  physical  representation  of  a 


problem  to  verify  the  correctness  of  their  method  and 
result,  e.g.,  by  checking  whether  all  forces  are 
balanced,  whether  all  entities  in  the  diagram  are 
related  to  givens  in  the  problem,  etc.  Similarly,  in 
chess,  Simon  ( 1972)  observed  that  some  masters  search 
the  space  of  future  moves  and  countermoves  to  verify 
that  the  moves  they  recognized  as  best  are  in  fact  in  the 
subset  of  good  moves.  More  recent  research  has  found 
that  differences  in  search  skill  (i.e.,  depth,  breadth,  and 
speed)  are  correlated  with  chess  expertise  (Charness, 
1981:  Holding  and  Reynolds,  1982).  Key  aspects  of 
searching  to  verify  recognized  answers  are 
metacognitive:  the  processes  of  initiating  search, 
monitoring  and  evaluating  its  results,  and  deciding 
when  it  should  be  terminated. 

In  sum,  metacognitive  processes  are  crucial  in  two 
phases  of  intuitive  decision  making: 

•  Constructing  of  a  situation  model  or  plan  when 
recognition  is  uncertain. 

•  Verifying  the  results  of  recognition 

We  have  devised  a  model  that  incorporates  both 
recognition  and  facilitative  processes,  and  we  call  it  the 
Recognition/Metacognition  model  (Cohen,  1993a).  It 
has  three  key  components  (see  figure  #2):  Quick  Test, 
Critiquing,  and  Correcting.  Each  represents  a  different 
category  of  skill  in  situation  understanding  and 
decision  making.  We  will  address  each  in  turn. 


2.5  Metacognitive  knowledge:  The  Quick  Test 

The  Quick  Test  is  a  gate-keeping  function  that 
determines  whether  ( 1 )  to  engage  critiquing  and 
correcting  processes  that  might  improve  problem 
recognition  or  (2)  whether  the  current  level  of 
recognition  can  (or  must)  suffice. 

This  process  answers  the  question:  Is  there  some 
reason  to  think  more  about  my  current  model  or  plan, 
or  should  I  act  immediately?  The  answer  is  based  on 
three  more  specific  questions:  ( 1 )  Do  I  have  time 
before  it  is  necessary  to  commit  to  a  decision?  (2)  Are 
the  stakes  of  an  error  high?  and  (3)  Is  there  significant 
reason  to  doubt  my  initial  situation  assessment  or  plan? 
Quick  Test  skills  thus  involve  sensitivity  to  the 
availability  of  time  and  potential  costs  of  delay; 
sensitivity  to  the  costs  of  errors  that  might  occur  if  one 
does  not  delay,  but  invokes  the  best  solution  recognized 
to  date;  and  sensitivity  to  the  typicality  of  a  situation 
and,  conversely,  to  the  presence  of  unusual  or  troubling 
features.  If  all  three  of  the  Quick  Test  conditions  are 
satisfied.  Quick  Test  inhibits  the  recognition-based 
response  and  triggers  a  process  of  knowledge-based 
reasoning.  If  at  least  one  of  the  Quick  Test  conditions 
is  not  satisfied,  the  initial  model  or  plan  is  accepted, 
and  no  critiquing  or  correcting  takes  place. 


Situation 

Model 


QUICK  TEST 


Real  World 


verity  (1)  Do  I  have  some  time  before  I  must 

>  commit  to  a  decision? 

(2)  Are  stakes  of  an  error  high? 
Verify  (3)  Is  the  situation  unfamiliar,  atypical? 
/  Or  is  there  some  specific  problem  with 
the  model  or  plan? 


-No  to  1,2,  or  3- 


Yes  to  1,  2,  &  3 


Figure  2:  Metacognitive  knowledge  and  process  flow 

The  adaptive  model  integrates  concepts  from  other 
models.  According  to  Klein  ( 1993),  rapid  recognition- 
primed  decision  making  is  expected  under  conditions 
of  high  time  pressure.  According  to  Connolly  and 
Wagner  (1988),  rapid  processing  may  occur  when 
there  is  low  cost  of  an  error.  According  to  both  Klein 
and  Rasmussen  ( 1993  ),  it  is  expected  in  highly  familiar 
situations,  that  is  from  decision  makers  with  situation- 
specific  expertise.  If  any  one  of  these  conditions  is 
true,  then  the  answer  to  the  corresponding  Quick  Test 
question  is  “no,”  and  correcting  and  critiquing  do  not 
take  place. 


The  Quick  Test  can  be  a  relatively  explicit  and 
conscious  process,  or  a  form  of  recognitional 
processing  at  a  higher  level  that  can  be  extremely  rapid 
and  virtually  automatic. 

2.6  Metacognitive  knowledge:  Critiquing 

An  initial  situation  model  or  plan  may  fail  the 
Quick  Test  because  specific  problems  are  apparent  or 
simply  because  the  situation  is  complex  or  unfamiliar. 
In  the  latter  case,  the  next  goal  in  metacognitive 
processing  is  to  answer  the  question:  Are  there  specific 
potential  problems  with  the  current  model  or  plan? 
Critiquing  is  the  stage  of  ferreting  out  such  problems. 


Critiquing  can  result  in  the  discovery  of  three  kinds 
of  problems  in  the  current  model  or  plan: 
incompleteness,  unreliability,  or  conflict.  Situation 
understanding  or  planning  is  incomplete  if  conclusions 
and  options  are  not  specific  and  detailed  enough. 
Understanding  and  planning  may  be  complete  but 
unreliable  if  the  link  between  data  and  evidence,  or  the 
link  between  actions  and  goals,  is  uncertain  or 
conditional.  Finally,  even  if  understanding  and 
planning  are  complete  and  free  of  obvious  unreliable 
assumptions,  there  may  be  alternative,  conflicting 
conclusions  that  better  account  for  some  of  the  data,  or 
alternative  actions  that  better  achieve  some  of  the 
goals. 

Some  critiquing  methods  are  general-purpose.  They 
are  capable  of  uncovering  problems  of  all  three  kinds. 
Mental  simulation  and  retrieval  of  similar  cases  from 
experience  are  general-purpose  in  this  sense.  For 
example,  by  mentally  simulating  a  course  of  action,  an 
officer  might  be  able  to  see  if  the  current  plan  has  any 
gaps,  if  it  reliably  achieves  goals,  and  whether  it 
conflicts  with  other  goals.  Similarly,  comparison  of  a 
plan  with  an  analogous  previous  experience  might 
reveal  gaps  in  the  present  plan,  suggest  places  where 
the  plan  might  not  work  reliably,  or  suggest  alternative 
actions  that  have  been  adopted  in  the  past. 

Other  critiquing  methods  are  more  specialized  in  the 
kinds  of  problems  they  can  uncover.  For  example, 
decision  makers  might  use  a  checklist  or  standard 
operating  procedure  to  ensure  that  all  required 
components  of  a  model  or  plan  have  been  specified 
(completeness).  They  might  adopt  a  devil’s  advocate 
technique  in  order  to  ferret  out  unreliable  assumptions 
in  an  assessment  or  prediction.  Data  collection  can 
determine  if  observations  are  consistent  or  conflicting 
with  the  current  situation  model  or  plan. 

2.7  Metacognitive  knowledge:  Correcting 

If  no  specific  problem  with  the  model  or  plan  is 
identified  by  either  quick  verification  or  critiquing, 
then  metacognitive  processing  in  the  current  cycle  is 
complete.  But  if  a  specific  problem  is  found,  the  third 
major  function  of  metacognition  is  enlisted:  facilitating 
the  construction  of  an  improved  model  or  plan. 
Whatever  problem  is  discovered,  three  methods  are 
available  to  solve  it: 

1 .  Collecting  more  data  to  fill  gaps  in  the  model  or 
plan,  confirm  or  disconfirm  an  assumption,  or  to 
resolve  conflict 

2.  Activating  existing  knowledge  in  long-term 
memory,  for  the  same  purposes 

3 .  Adding  assumptions  to  fill  gaps  or  resolve  conflict, 
and  dropping  assumptions  when  they  appear 
unreliable 

Metacognitive  processes  play  a  role  in  choosing 
among  these  processes,  and  in  regulating  the  process 


that  is  chosen:  (1)  in  selecting  the  amount  and  type  of 
data  collection,  (2)  in  directing  the  search  for 
knowledge  in  long-term  memory,  and  (3)  in  adjudicat¬ 
ing  among  competing  possible  assumptions. 

Data  collection.  Sometimes  there  is  time  and 
opportunity  to  collect  additional  data  to  flesh  out  or 
resolve  ambiguity  in  a  model  or  plan,  or  confirm  or 
disconfirm  doubtful  assumptions.  The  decision  to 
collect  more  data  rather  than  simply  think  about  the 
problem  involves  metacognitive  judgments  regarding 
the  amount  of  available  time,  the  cost  and  potential 
risks  of  data  collection,  and  the  trustworthiness  of 
information  sources. 

Knowledge  activation.  Metacognitive  processes 
are  crucial  in  guiding  the  serial  activation  of 
knowledge  in  long-term  memory.  Such  searches  are 
conceived  of  as  controlled  spreading  activation. 
Executive  processes  determine  which  components  of 
the  current  model  will  be  attended,  thus  influencing  the 
portions  of  long-term  memory  likely  to  be  activated 
next  (McClelland  &  Rumelhart,  1986).  The  values  of 
the  attended  nodes  are  fixed,  or  clamped,  at  a  high 
level  of  activation  (in  effect,  accepting  them 
provisionally  or  by  assumption)  in  order  to  explore 
their  implications.  In  the  next  cycle,  new  nodes  may  be 
clamped,  and  so  on,  until  knowledge  is  activated  that 
satisfies  the  goals  of  the  search  (or  quick  verification 
determines  that  time  has  run  out).  Different  officers 
will  attend  to  aspects  of  the  situation  in  different 
orders.  Some  may  focus  attention  on  knowledge  of 
terrain,  others  on  knowledge  of  enemy  strength,  others 
on  knowledge  about  enemy  goals,  and  others  on 
knowledge  of  enemy  actions. 

Metacognitive  control  may  influence  search  in 
another  way,  by  determining  its  temperature  ( Flinton  & 
Sejnowski,  1986),  i.e.,  by  adjusting  the  degree  of 
similarity  required  for  a  match  between  patterns  in 
active  memory  and  stored  structures.  At  high 
temperatures,  the  activation  net  is  cast  wide,  and  far¬ 
fetched  ideas  have  a  significant  chance  of  being 
considered.  At  low  temperatures,  an  idea  must  have  a 
very  high  degree  of  association  with  currently  active 
beliefs  to  have  a  chance  of  being  activated.  High 
temperatures  may  be  crucial,  for  example,  when  no 
plan  is  readily  available  that  can  achieve  important 
goals. 

Adjusting  assumptions.  If  data  collection  is 
infeasible  because  of  limitations  in  resources,  time,  or 
sources  of  information,  and  if  definitive  knowledge  is 
not  available  or  cannot  be  accessed  from  long-term 
memory,  the  situation  assessor  may  revise  his 
interpretation  of  the  information.  Metacognitive 
processes  are  crucial  in  the  interpretative  process  of 
evaluating  and  revising  assumptions.  Decision 
makers  think  and  act  as  if  assumptions  were  true  until 
there  is  some  reason  to  doubt  them.  Conflict  between 


data  and  a  situation  model,  or  between  two  competing 
models,  provides  such  a  reason  for  doubt.  Conflict 
indicates  that  at  least  one  of  the  beliefs  involved  in 
building  the  models  or  interpreting  the  data  was  false. 
Conflict  may  thus  trigger  a  metacognitive  process  of 
exposing  and  questioning  assumptions.  The  process 
of  revising  beliefs  to  explain  conflict  requires  a  variety 
of  metacognitive  skills:  awareness  that  conflict  exists, 
an  ability  to  uncover  implicit  assumptions  that  have 
created  the  conflict,  sufficient  awareness  of  the 
structure  of  one's  beliefs  to  identify  the  assumptions 
that  are  central  to  a  variety  of  models  and  plans,  and 
recall  of  past  episodes  in  which  the  same  beliefs  may 
have  led  to  a  conflict.  Finally,  the  process  of 
assumption  revision  calls  for  balancing  the  plausibility 
and  the  power  of  the  resulting  models  and  plans. 

3.  Teaching  Critiquing  Skills 

3.1  Introduction 

The  simplest  description  of  the  training 
implications  of  the  adaptive  model  involves  contrasting 
them  with  the  implications  of  the  analytical  and 
recognitional  models. 

If  we  view  decision  making  as  an  analytical 
process,  a  perfectly  appropriate  stance  in  some 
circumstances,  then  it  follows  that  instruction  should 
focus  on  the  transformations  required  to  represent 
problems  within  the  analytical  model  (for  example, 
quantitative  representations).  Practice  may  be 
partitioned  such  that  trainees  study  parts  of  the  method 
before  attempting  to  execute  the  whole.  Examples 
should  be  varied  in  character  to  demonstrate  the 
generality  of  the  method,  and  they  should  be  presented 
in  a  graduated  sequence  progressing  from  problems 
that  clearly  afford  the  analytical  technique  (that  is,  they 
require  little  in  the  way  of  transformation)  to  those  that 
superficially  seem  inappropriate  to  the  technique  (that 
is,  they  are  quite  difficult  to  completely  and  accurately 
represent). 

If  decision  making  is  taken  to  be  a  recognitional 
process,  again  a  reasonable  assumption  in  some 
situations,  then  the  following  inferences  about  training 
hold.  Instruction  should  focus  on  goals,  environmental 
conditions,  and  actions.  In  procedural  terms, 
instruction  should  convey  the  following  type  of 
knowledge:  if  your  goal  is  X  amidst  events  Y,  then 
perform  action  Z.  Practice  conditions  should  be 
realistic  to  promote  accurate  recognition  and  repetitive 
to  facilitate  automatic  performance.  Examples  should 
be  representative  of  the  domain,  though  feedback  may 
be  manipulated  to  speed  training. 

Training  that  assumes  an  adaptive  model  of 
decision  making  should  first  of  all  convey  the  iterative 
nature  of  problem  solving  —  from  recognition  to 
metacognitive  checks,  critiques,  and  corrections,  back 


to  recognition.  It  must  also  aim  to  sensitize  trainees  to 
domain-specific  cues  concerning  the  time  constraints, 
stakes,  and  familiarity  of  problems,  as  well  as  the 
nature  of  conflict,  completeness  and  reliability  in 
specific  domains.  It  must  convey  methods  of  critiquing 
situation  assessments  and  plans,  and  of  making 
corrections  that  will  enhance  recognition.  Practice 
should  be  designed  to  make  metacognitive  processes 
explicit.  Such  practice  might  involve  reciprocal 
teaching  (Palincsar  and  Brown,  1984),  team  exercises, 
explicit  labeling  of  one’s  own  activities,  listening  to 
experts  label  their  thought  processes,  or  critiquing  the 
performance  of  peers.  All  of  these  methods  help 
trainees  make  public  otherwise  hidden  cognitive 
activities.  Making  principles  explicit  may  helps 
trainees  transfer  what  they  have  learned  to  varied 
settings  (Collins,  Brown,  &  Newman,  1989).  In 
addition,  it  may  be  appropriate  to  give  trainees  control 
over  practice  parameters  that  cue  metacognitive 
activities;  these  include  time,  immediacy  of  feedback, 
quality  of  feedback,  and  difficulty.  Exercises  should 
employ  non-routine  cases,  as  these  discourage  simple 
recognitional  solution  and  encourage  the  use  of 
metacognitive  skills. 

We  have  attempted  to  bear  these  guidelines  in  mind 
in  an  ongoing  study  of  the  efficacy  of  training 
metacognitive  skills.  That  training  focuses  on  one 
metacognitive  skill:  critiquing  situation  assessments. 

3.2  A  course  in  critiquing  skills 

In  teaching  critiquing,  we  attempt  to  unify  the  three 
potential  weaknesses  of  assessments  (incompleteness, 
unreliability,  and  conflicting  evidence)  under  a  single 
concept:  the  plausibility  of  the  assessment. 

The  analytical  approach  to  plausibility  is  numerical 
assessment.  The  meaning  of  a  piece  of  information  for 
a  particular  hypothesis  is  summarized  in  a  single 
number  (e.g.,  a  probability  or  a  belief  function) 
measuring  its  diagnostic  impact.  The  result  is  an 
average  that  suffers  two  failings.  The  first  is  that 
quantitative  solutions  (e.g.,  such  as  hostile  intent  =  .3; 
probability  not  hostile  intent  =  .7)  are  inherently 
abstract,  and  thus  difficult  to  comprehend.  The  second 
is  that  the  coherence  of  the  evidence  underlying  the 
average  cannot  be  communicated  in  a  natural, 
conversational  manner  (a  requisite  condition  of  good 
performance  in  a  team  setting),  but  only  by 
reconstructing  the  complex  analytical  process. 

In  simple  recognitional  approaches,  plausibility  is 
represented  by  the  degree  of  match  or  mismatch 
between  a  complete  situation  and  a  pre-stored  template. 
The  result  is  typically  a  single  best-matching  template. 
However,  novel  situations  are  liable  to  produce  gross 
mismatches.  Since  recognitional  processes  are  not, 
presumably,  available  for  direct  verbal  report  or 
analysis,  the  extent  and  cause  of  these  mismatches 


cannot  be  explained,  nor  can  alternative  matches  be 
made  explicit. 

In  the  adaptive  framework  the  end  result  is  a  single 
coherent  situation  model  ( an  elaborated  schema  for  a 
given  situation).  This  distinguishes  the  adaptive 
framework  from  the  analytical  approach  but  is  quite 
similar  to  the  product  of  recognitional  processing. 
However,  the  process  of  constructing  the  schema  is 
explicit  in  the  adaptive  approach,  unlike  the 
recognitional  process.  Thus,  the  process  is  available 
for  presentation  and  public  scrutiny  in  a  natural, 
conversational  context.  In  particular,  the  situation 
assessor  is  able  to  articulate  the  assumptions 
underlying  his  or  her  assessment. 

Metacognitive  strategies  for  critiquing  involve 
distinguishing  between  the  normal  interpretation  of  a 
cue  and  possible  exceptions  (see  Toulmin,  1978).  The 
initial  recognitional  response  to  a  cue  provides  its 
normal  meaning.  For  example,  the  absence  of  enemy 
artillery  within  shelling  range  may  indicate  that  an 
encampment  site  is  safe  from  shelling.  Exceptions  to 
this  normal  meaning  can  be  generated  by  a  method 
discussed  in  the  next  section  when  time  is  available, 
stakes  are  high,  and  the  situation  is  unfamiliar. 

Perhaps  the  enemy  is  prepared  to  move  artillery 
quickly  within  range,  our  scouts  did  not  find  an 
existing  artillery  site,  or  the  enemy  plans  to  use  air 
assault.  These  exceptions  constitute  assumptions  that 
must  be  false  if  the  assessment  of  a  safe  encampment  is 
true.  However,  the  decision  maker  may  never  actually 
have  recognized  and  discarded  these  assumptions  in 
formulating  the  assessment.  Metacognitive  skills 
training  may  help  officers  uncover  hidden  assumptions 
early. 

The  same  skills  used  to  critique  situation 
assessments  can  be  extended  to  analyzing  plans.  Thus, 
critiquing  skills  can  help  an  officer  identify  hidden 
assumptions  in  plans,  and  deal  with  them  in  a  variety 
of  ways.  The  decision  maker  may  decide  that  an 
assumption  is  plausible  on  the  basis  of  recalled 
evidence,  confirm  its  truth  by  examining  extant  data  or 
requesting  intel,  take  steps  to  make  the  assumption  true 
through  proactive  strategies,  adopt  a  contingency  plan 
in  case  the  assumption  is  false,  develop  a  model  or  plan 
that  does  not  depend  on  the  assumption  at  all,  or  accept 
the  assumption  as  a  known  risk. 

In  an  ongoing  study  we  provide  training  in  two 
metacognitive  skills.  The  first  method  is  designed  to 
help  officers  critique  assessments  or  plans  to  find 
hidden  assumptions  before  they  cause  problems.  In 
essence,  the  method  helps  counteract  overconfidence. 
The  second  method  is  designed  to  help  officers  refine 
or  change  their  assessments  after  hidden  assumptions, 
inadequate  plans,  or  other  causes  produce  problems.  It 
is  a  cure  for  underconfidence,  or  confusion. 


3.3  Finding  Hidden  Assumptions 

The  method  of  finding  hidden  assumptions  consists 
of  four  steps. 

1 .  Select  a  critical  part  of  your  assessment  —  even  if 
you  are  confident  of  it. 

2.  Imagine  that  a  “perfect"  intelligence  source  (such 
as  a  crystal  ball)  tells  you  that  this  part  of  your 
assessment  is  wrong. 

3.  Explain  how  it  could  be  wrong 

4.  The  “perfect”  intelligence  source  now  tells  you 
that  this  explanation  is  wrong... go  back  to  step  3. 

The  product  of  this  exercise  is  a  list  of  negations  of 
assumptions  that  underlie  the  assessment.  For  the 
assessment  to  stand,  the  officer  must  demonstrate  that 
the  assumptions  are  unreasonable,  that  is,  they  cannot 
plausibly  be  used  to  generate  an  opposing  assessment. 
The  following  illustration  —  drawn  from  an  interactive 
classroom  exercise  at  Ft.  Lewis  —  may  clarify  this 
point. 

Suppose  that  an  officer's  assessment  includes  the 
claim  that  the  enemy  will  cross  the  river  at  location  X. 
He  bases  this  claim  on  arguments  concerning  the 
distance  the  enemy  must  travel  to  his  supposed 
objective,  the  shallow  depth  of  the  river,  and 
concealment  opportunities  along  the  bank.  The  officer 
is  confident  of  this  assessment;  however  stakes  are 
high  and  there  is  time  to  critique  the  assessment,  so  he 
does  so.  Following  the  method,  above,  he  imagines 
that  a  “perfect”  intelligence  source  tells  him  that  the 
enemy  will  not  cross  at  location  X,  and  demands  that 
he  explain  this  failure  in  his  interpretation  of  the 
evidence.  He  cycles  through  steps  three  and  four  to 
generate  the  following  list  of  assumptions: 

•  “The  enemy  anticipates  that  our  force  will  be  at 
location  X.” 

•  “The  enemy  will  detect  the  movement  of  our  force 
to  location  X.” 

•  “There  are  good  crossing  sites  that  we  missed.” 

•  “The  enemy  doesn't  know  how  good  a  location  X 
is.” 

•  “The  enemy  doesn't  have  any  river  crossing  assets. 
He  can't  cross  the  river  at  all.” 

•  “The  enemy’s  river  crossing  assets  are  so  good 
that  he  can  cross  elsewhere.” 

•  “The  enemy  has  a  large  enough  force  that  he  can 
accept  casualties  crossing  elsewhere.” 

•  “The  enemy’s  objectives  are  different.  He  doesn't 
need  to  cross  at  all.” 

•  “The  enemy  will  use  air  assault  to  get  across  the 
river,  rather  than  cross  it.” 

These  nine  ideas  are  exceptions  to  the  officer’s 
recognitional  assessment  of  the  situation.  To  maintain 
the  assessment  that  the  enemy  will  cross  at  location  X, 
the  officer  must  prove  to  himself  that  it  is  reasonable  to 
assume  they  are  false  or  that  he  has  some  way  to 
handle  the  situation  if  they  are  true. 


Table  #1  includes  several  of  the  types  of  responses 
that  were  discussed  above:  testing  the  truth  of  an 
assumption  through  recall  of  evidence  or  data 
collection,  adopting  proactive  strategies,  adopting 
contingency  plans,  and  accepting  the  assumption  as  a 
known  risk. 

We  have  found  this  method  to  be  fast  (usually  less 
than  five  minutes  in  duration)  and  highly  productive. 

It  is  particularly  effective  for  officers  when  they  are 
confident  of  their  assessments.  It  raises  issues  that 
force  officers  to  earn  their  own  confidence  by 
strengthening  their  assessments  and  plans  to 
compensate  for  potential  weaknesses. 

Exceptions 

Responses 

“The  enemy  anticipates  that  our  force  will  be  at  location 
X.” 

Place  our  forces  elsewhere  then  move. 

“The  enemy  will  detect  the  movement  of  our  force  to 
location  X.” 

Use  deception. 

“There  are  good  crossing  sites  that  we  missed.” 

Consult  with  a  specialist  in  river  crossings  or  scout 
enemy  movements  towards  other  crossing  sites. 

“The  enemy  doesn't  know  how  good  a  location  X  is.” 

Dismiss  this  as  implausible  given  recent  intel  on  enemy 
surveying  and  scouting  activity. 

“The  enemy  doesn't  have  any  river  crossing  assets.  He 
can't  cross  the  river  at  all.” 

Dismiss  this.  It  presents  no  problem. 

“The  enemy’s  river  crossing  assets  are  so  good  that  he 
can  cross  elsewhere.” 

Collect  intel  to  verify  or  disprove. 

“The  enemy  has  a  large  enough  force  that  he  can  accept 
casualties  crossing  elsewhere.” 

Dismiss  this  as  implausible  given  current,  reliable  intel. 

“The  enemy’s  objectives  are  different.  He  doesn't  need 
to  cross  at  all.” 

Adopt  a  contingency  plan. 

“The  enemy  will  use  air  assault  to  get  across  the  river, 
rather  than  cross  it.” 

Accept  this  risk. 

Table  1:  Example  of  exceptions  to  the  interpretation  of  evidence,  and  methods  of  handling  the  exceptions. 


productive.  It  instructs  the  officer  to  imagine  that  the 
assessment  is  true,  and  that  he  must  explain  the 
surprising  event.  Trainees  often  find  it  useful  to  focus 
the  technique  by  specifying  as  the  domain  of 
explanations  the  categories  of  METT-T  (mission, 
enemy,  troops,  terrain  and  weather,  and  time 
available).  For  example,  the  trainee  may  consider  how 
an  event  might  be  explained  in  terms  of  actions  of  his 
own  troops,  the  terrain  or  weather,  enemy  tactics, 
enemy  equipment,  and  so  forth.  As  discussed  above, 
the  product  of  these  methods  is  an  assumption  that 
must  be  verified,  planned  against,  or  handled  in  some 
other  way. 

Third,  the  coherence  of  the  assessment  is  a  function 
of  the  most  plausible  union  of  explanations  generated 
to  account  for  unexpected  events.  An  example  may 
clarify  this  point.  It  is  presented  here  in  a  table  format 
used  during  training  (see  table  #2).  The  table  shows  an 
assessment  and  three  events  that  are  surprising  in  light 
of  the  assessment.  The  decision  maker  has  used  the 
perfect  intelligence  source  to  generate  a  set  of 
explanations  for  each  event.  The  METT-T  category 
used  to  generate  each  explanation  is  presented  in 
brackets.  Explanations  marked  with  an  asterisk  are 
those  the  officer  considers  the  most  plausible  of  the 

explanations.  The  “perfect”  information  source  is 


3.4  Handling  the  Unexpected 

Despite  expert  assessment,  careful  planning,  and 
thorough  critiquing,  officers  often  are  confronted  by 
unexpected  events.  The  second  method  in  which  we 
have  trained  officers  is  designed  to  help  them  handle 
the  unexpected,  either  by  explaining  events  in  terms  of 
their  current  assessment,  or,  if  explanations  are 
implausible,  by  altering  their  assessments  to  conform 
to  the  new  state  of  the  world.  The  procedure  consists 
of  these  steps: 

1 .  Notice  an  unexpected  event. 

2.  Try  to  explain  the  event  in  terms  of  your  current 
assessment. 

3.  Evaluate  the  plausibility  of  your  explanations  for 
all  unexpected  events  to  date. 

4.  If  the  explanations  are  not  plausible,  change  your 
assessment 

Several  caveats  and  tips  pertain  to  this  method. 
First,  it  is  sometimes  enough  simply  to  notice  an 
unexpected  event.  This  is  the  case  if  there  is  no  time  to 
analyze  the  event,  if  the  event  does  not  raise  the  stakes 
of  the  situation  above  some  threshold,  or  if  the  event 
does  not  lower  familiarity  with  the  situation. 

Second,  the  officer  can  use  two  tools  to  generate 


ones  he  generates  using  the  “perfect”  intelligence 
source. 

Note  that  the  best  overall  account  for  a  series  of 
unexpected  events  may  balance  simplicity  and 
plausibility.  For  example,  taken  separately,  the  most 
plausible  three  explanations  for  the  events  in  the 
example  concern  troops,  enemy  deception,  and  enemy 
tactics.  However,  it  may  be  improbable  to  an  officer 


Assessment:  The  enemy  plans  a  diversionary  attack  in 
the  north,  and  a  main  attack  in  the  south. 

Events 

Explanations 

Southern  enemy  force  has  destroyed  a  major  bridge  to 
his  front 

*  [Troops]  Perhaps  it  was  our  own  forces,  not  the  enemy, 
that  destroyed  the  bridge. 

[Enemy  (deception)]  Perhaps  the  destruction  is  a 
deception  to  make  you  think  the  main  attack  will  not  be 
in  the  southern  sector. 

[Enemy  (mistake)]  Perhaps  the  destruction  of  the  bridge 
was  a  mistake  by  the  enemy. 

Enemy  has  initiated  radio  silence  in  the  north  and  the 
south. 

[Troops]  The  enemy  is  not  responsible  for  radio  silence. 
Our  interdiction  campaign  may  have  destroyed  critical 
enemy  radio  facilities. 

*  [Enemy  (deception)]  The  enemy  is  concealing  the 
location  of  the  attack  by  instituting  radio  silence 
everywhere. 

[Enemy  (equipment)]  The  enemy’s  C2  equipment  has 
failed. 

Reinforcements  have  arrived  in  the  north. 

[Enemy  (deception)]  The  enemy  is  “showing”  arrival  in 
the  north  before  shifting  forces  stealthily  to  the  south. 

*  [Enemy  (tactics)]  The  enemy  may  be  putting  green 
forces  on  line  in  the  north  to  execute  a  secondary  attack. 
[Enemy  (mistake)]  The  northern  movement  of 
reinforcements  may  be  a  mistake. 

Table  2:  Table  format  used  to  train  both  and  finding  hidden  assumptions  and  handling  the  unexpected. 


that  all  three  of  these  explanations  are  independently 
true.  Thus,  for  example,  he  may  choose  a  somewhat 
simpler  account,  in  which  a  single  cause  —  our  own 
aggressive  interdiction  campaign  —  accounts  for  the 
destruction  of  bridges  and  radio  silence,  and  a  second 
assumption  concerning  enemy  tactics  explains 
reinforcements  in  the  north. 


An  officer  may  eventually  find  that  he  cannot 
construct  a  coherent  account  of  all  of  the  unexpected 
events.  There  may  be  just  one  too  many  events  to 
explain.  Suppose  that  the  example  above  were 
extended  to  include  the  surprising  movement  of  two 
enemy  motorized  rifle  regiments  from  the  south  to  the 
north.  Then  it  might  be  the  case  that  no  additional 
explanation  (  such  as  enemy  deception  or  errors)  could 
make  tenable  the  assessment  that  the  enemy  plans  a 
main  attack  in  the  south  and  a  diversionary  attack  in 
the  north.  In  that  case,  a  new  assessment  is  in  order, 
per  the  fourth  step  of  the  procedure  outlined  above. 

We  teach  the  process  of  generating  and  justifying  a 
new  assessment  using  the  same  type  of  table  employed 
above.  The  process  begins  by  examining  the 
unexpected  events  to  date  (ignoring  explanations  of 
their  meaning)  and  arriving  at  an  assessment  of  them. 
This  process  can  be  recognitional.  To  tap  this 
potential,  the  trainee  is  instructed  to  answer  the 
question.  “What  do  these  events  usually  mean?”  In  the 
case  above,  that  interpretation  might  be  that  the  enemy 


plans  its  main  attack  in  the  north.  A  new  table  similar 
to  table  #2  is  then  constructed,  and  the  events  column 
is  filled  with  evidence  that  supported  the  original 
assessment  (now  discarded).  Standard  interpretations 
of  these  events  would  probably  contradict  the  new 
assessment.  Thus,  these  events  must  be  explained  in  a 
manner  that  is  consistent  with  the  new  assessment. 

The  list  of  unexpected  events  and  their  explanations 
grows  as  surprises  arise,  until  the  assessment  is  again 
replaced  or  the  situation  is  resolved. 

3.5  Status  Report  on  a  Pilot  Study 

We  have  taught  these  techniques  to  33  officers,  thus 
far.  Trainees  perform  a  pretest,  receive  the  training, 
and  take  a  posttest.  The  pretest  and  posttest  materials 
are  counterbalanced  to  control  for  possible  differences 
in  the  difficulty  of  materials.  A  control  group  receives 
a  pretest,  discusses  military  tactics,  and  completes  a 
posttest.  This  aspect  of  the  design  controls  for  the 
effect  of  practice  during  training. 


Training  for  the  experimental  group  consists  of 
reading  a  short  illustrated  manual,  summary  lectures 
by  the  instructor,  and  interactive  exercises  in  which 
officers  critique  situations  from  their  own  experience, 
and  assessments  they  have  made. 

The  test  materials  consist  of  a  military  scenario  and 
a  dozen  problem  statements.  The  scenario  describes 
the  invasion  of  an  island  nation  by  its  neighbor  and  the 
first  actions  taken  by  the  U.S.  in  response.  It  consists 
of  a  chronologically  organized  status  report,  a  mission 
statement,  and  a  map  of  the  island.  The  problem 
statements  consist  of  two  parts,  each  printed  on  a 
separate  page.  The  first  part  contains  background 
information  ( 150  words,  on  average)  concerning  a 
particular  topic  (e.g.,  the  status  of  hostages  held  by 
enemy  forces,  the  air  defense  equipment  and  readiness 
of  the  enemy)  and  an  assessment  of  the  situation 
regarding  that  particular  topic.  The  second  part 
consists  of  identical  text  plus  new  information 
concerning  the  topic  (not  a  new  assessment). 

On  each  part  of  each  problem,  subjects  are  asked  to 
perform  two  tasks,  phrased  as  follows: 

•  Please  evaluate  the  assessment.  In  what  ways  is 
the  reasoning  good?  bad? 

•  Do  you  agree  with  the  assessment?  Use  this  scale: 

1 .  Strongly  disagree 

2.  Moderately  disagree 

3.  Don't  know 

4.  Moderately  agree 

5.  Strongly  agree 

Six  of  the  twelve  problems  appear  on  the  pretest 
and  six  on  the  posttest.  Answer  sheets  consist  of  one 
piece  of  blank  paper  per  problem,  labeled  with  the 
problem  number. 

The  study  and  preliminary  analysis  of  its  results  are 
in  progress.  Thus,  far,  we  can  report  that  corps 
planners  and  other  officers  at  the  level  of  Captain, 
Major,  and  Lt.  Colonel  find  the  training  useful,  and 
ask  that  the  presentation  be  lengthened  from  the 
current  two  hours  to  a  day  or  more.  Subjective 
assessments  aside,  we  use  as  dependent  measures  the 
quantity  and  quality  of  issues  that  trainees  raise  in  their 
evaluations  of  the  printed  assessments.  Quality  is 
assessed  by  expert  judges  blind  to  the  test  conditions. 
However,  we  are  not  satisfied  that  this  approach  will 
net  strong  results.  It  is  a  significant  challenge  to  effect 
and  detect  differences  in  the  use  of  briefly  trained  skills 
on  complex  cognitive  tasks. 

4.  Conclusion 

We  have  explored  only  part  of  the  potential 
curriculum  in  metacognitive  skills,  and  we  have  used 
only  a  few  of  the  potential  training  techniques  to  do  so. 
Here  we  wish  to  raise  some  of  the  questions  that 
pertain  to  future  research. 


Skills  relevant  to  assessing  the  time  and  stakes  of  a 
situation  (part  of  what  we  call  the  Quick  Test)  may 
require  training  that  hones  recognitional  processing  of 
the  tempo  and  dangers  of  specific  scenarios.  We  are 
interested  in  investigating  how  recognitional  training 
can  be  structured  to  help  officers  generalize  the  lessons 
learned  from  instruction  that  is  so  specific  in  content. 
The  issue  of  generalization  also  arises  in  considering 
how  to  train  the  third  component  of  the  Quick  Test: 
metacognitive  sensitivity  to  the  familiarity  of 
situations. 

Many  of  the  skills  needed  to  correct  assessments 
and  plans  may  already  be  well  taught  by  the  Army.  We 
are  thinking  here  of  such  skills  as  developing 
contingency  plans.  However,  it  would  be  useful  to 
study  how  to  help  officers  link  the  outcomes  of 
metacognitive  critiquing  processes  to  their  current 
skills,  and  to  investigate  which  correction  skills  are  not 
yet  addressed  in  the  Army  curriculum. 

At  a  more  general  level,  we  are  concerned  with  the 
manner  in  which  the  metacognitive  skills  used  by 
individuals  interact  with  the  social  processes  employed 
by  groups  of  officers  who  formulate  and  critique 
assessments. 

The  representation  of  knowledge  is  a  significant 
issue  at  the  levels  of  the  individual  and  the  group.  For 
example,  we  are  concerned  with  how  schemas  and 
situation  models  can  best  be  represented  for  individual 
use  in  training  and  in  the  field.  It  is  also  unclear 
whether  the  representations  that  best  facilitate  the 
thinking  of  individuals  will  optimally  support  group 
interactions. 

These  and  other  issues  remain  to  be  systematically 
addressed  in  naturalistic  settings.  It  is  our  hope  that 
studies  of  training  interventions  may  help  Army 
instructors  devise  courses  in  metacognitive  skills, 
courses  that  are  accurate  in  content  and  principled  in 
method. 

Finally,  we  believe  that  the  Recognition/ 
Metacognition  model  also  has  implications  for  the 
design  of  decision  support  systems  and  man-machine 
interactions  in  the  Army  environment.  We  have  begun 
to  assess  its  potential  in  these  areas. 
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